A Method for Large-Scale 1-Regularized Logistic Regression

Logistic regression with 1 regularization has been proposed as a promising method for feature selection in classification problems. Several specialized solution methods have been proposed for 1-regularized logistic regression problems (LRPs). However, existing methods do not scale well to large problems that arise in many practical settings. In this paper we describe an efficient interior-point method for solving 1-regularized LRPs. Small problems with up to a thousand or so features and examples can be solved in seconds on a PC. A variation on the basic method, that uses a preconditioned conjugate gradient method to compute the search step, can solve large sparse problems, with a million features and examples (e.g., the 20 Newsgroups data set), in a few tens of minutes, on a PC. Numerical experiments show that our method outperforms standard methods for solving convex optimization problems as well as other methods specifically designed for 1regularized LRPs. Introduction Logistic regression Let x ∈ R denote a vector of feature variables, and b ∈ {−1,+1} denote the associated binary output. In the logistic model, the conditional probability of b, given x, has the form Prob(b|x) = 1/(1 + exp ( −b(wx+ v) ) ). The parameters of this model are v ∈ R (the intercept) and w ∈ R (the weight vector). Suppose we are given a set of training or observed examples, (xi, bi) ∈ R × {−1,+1}, i = 1, . . . ,m, assumed to be independent samples from a distribution. The model parameters w and v can be found by maximum likelihood estimation from the observed examples. The maximum likelihood estimate minimizes the average loss lavg(v, w) = (1/m) m ∑